Hi, I think it would be better to do just a documentation change and keep the current implementation.

I wrote the reasons into the issue, as the current wording of the issue is asking for a code change.

#1923 (comment)

Two comments from my side.

And regarding this:

Hi, I think it would be better to do just a documentation change and keep the current implementation.

I wrote the reasons into the issue, as the current wording of the issue is asking for a code change.

Exposing high-level convenience arguments on crawlers, which configure the underlying components, is a Crawlee design choice. And we follow this all over the place - crawlers (and Actor in SDK) already act as partial facades over the components they compose. A few examples:

• PlaywrightCrawler - headless, browser_type, browser_launch_options, use_incognito_pages, user_data_dir, and similar BrowserPool/plugin internals directly on the crawler. With your approach, there should be only browser_pool.
• StagehandCrawler follows the same pattern.
• BasicCrawler has use_session_pool: bool next to the session_pool object, which is the same shape as impersonate.

So impersonate is not introducing a new pattern. It follows the same approach we already use for browsers, sessions, proxy configuration, concurrency, and more.

This design decision was made a long time ago, and we should be consistent and follow it, rather than diverging from it. And AFAIK @B4nan is a strong proponent of this approach.

This creates the ugly edge case HttpCrawler(impersonate=False, http_client=...).

We should simply validate the argument combination and raise an error when it is invalid, like in other places.

...

TLDR; A convenience flag (with guard and documentation) is consistent with the rest of Crawlee. It also keeps the simple case simple: users can turn off impersonation without needing to know what an HTTP client is.